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METHOD, SYSTEM AND COMPUTER PROGRAM PRODUCT FOR MULTI- 
MODALITY REGISTRATION USING VIRTUAL CURSORS 

STATEMENT REGARDING FEDERALY SPONSORED RESEARCH AND 
DEVELOPMENT 

[0001] The government may have rights in this invention pursuant to 
Subcontract 22287 issued from the Office of Naval Research/Henry M. Jackson 
Foundation. 

BACKGROUND OF THE INVENTION 

[0002] The present disclosure relates generally to multi-modality registration 
and, more particularly, to a method of using virtual cursors to show pixel-to-pixel 
correspondence between multi-modality datasets. 

[0003] Recently, multi-modality acquisitions in the medical imaging field 
have become more common. Different modalities have different strengths and may 
provide unique diagnostic information. For example, when performing 
mammograms, ultrasound is particularly effective at differentiating benign cysts and 
masses while x-ray is typically used for detailed characterization of 
microcalcifications. Combining the images generated using an x-ray detector with 
with the images generated using an ultrasound system leverages the strengths of both 
modalities. However, one of the challenging aspects is how to spatially register these 
modality datasets, such that there is a one to one pixel/voxel correspondence while 
preserving the quality of the original scanned data. One of the challenges in 
registering 2D x-ray data to 3D ultrasound data is the magnification factor in the 
projected 2D x-ray data. This factor is calculated based on the spatial relationship 
between the x-ray source, the scanned object and the image receptor. Registering 3D 
ultrasound data to 2D x-ray data requires corrections for the magnification factor of 
the x-ray data in every 3D ultrasound slice because ultrasound data does not contain a 
magnification factor. Scaling the ultrasound data to account for the magnification 
factor increases the size of the ultrasound data and may produce or change the normal 
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look of the ultrasound data. The result of scaling the ultrasound image data is that the 
ultrasound data may have an appearance that looks different than the standard view 
that mammographers and radiologists are accustomed to viewing and analyzing. This 
may be especially pronounced when the x-ray source is not centered on the image 
receptor as is the case during the x-ray to ultrasound mammography data acquisition. 

BRIEF DESCRIPTION OF THE INVENTION 

[0004] In one aspect, a method for multi-modality registration using virtual 
cursors includes receiving a two-dimensional image dataset for an object at a first 
position and receiving a three-dimensional image dataset for the object at the first 
position. The three-dimensional image dataset includes a plurality of image slices. 
The two-dimensional image dataset is registered with the three-dimensional image 
dataset without taking into account a magnification factor. A user cursor position for 
a location in the two-dimensional image dataset is received. A slice of interest in the 
three-dimensional image dataset is received. The slice of interest is selected from the 
plurality of image slices in the three-dimensional image dataset. A shadow cursor 
position for a location in the three dimensional dataset is calculated. The shadow 
cursor position corresponds to the user cursor position and the calculating includes a 
correction for the magnification factor corresponding to the shadow cursor position 
for the slice of interest. The shadow cursor position is output. 

[0005] In another aspect, a method for multi-modality registration using 
virtual cursors includes receiving a two-dimensional image dataset for an object at a 
first position and receiving a three-dimensional image dataset for the object at the first 
position. The three-dimensional image dataset includes a plurality of image slices. 
The two-dimensional image dataset is registered with the three-dimensional image 
dataset without taking into account a magnification factor. A slice of interest in the 
three-dimensional image dataset is received. The slice of interest is selected from the 
plurality of image slices. A user cursor position for a location in the slice of interest 
in the three-dimensional image dataset is received. A shadow cursor position for a 
location in the two-dimensional image dataset is calculated. The shadow cursor 
position corresponds to the user cursor position and the calculating includes a 
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correction for the magnification factor corresponding to the shadow cursor position. 
The shadow cursor position is output. 

[0006] In still another aspect, a system for multi-modality registration using 
virtual cursors includes a computer system in communication with a first imaging 
system and a second imaging system. The first imaging system creates a two- 
dimensional image dataset for an object at a first position and the second imaging 
system creates a three-dimensional image dataset of the object at the first position. 
The three-dimensional image dataset includes a plurality of image slices. The 
computer system includes instructions to implement a method comprising receiving 
the two-dimensional image dataset from the first imaging system and receiving the 
three-dimensional image dataset from the second imaging system. The two- 
dimensional image dataset is registered with the three-dimensional image dataset 
without taking into account a magnification factor. A user cursor position for a 
location in the two-dimensional image dataset is received. A slice of interest in the 
three-dimensional image dataset is received. The slice of interest is selected from the 
plurality of image slices in the three-dimensional image dataset. A shadow cursor 
position for a location in the three dimensional dataset is calculated. The shadow 
cursor position corresponds to the user cursor position and the calculating includes a 
correction for the magnification factor corresponding to the shadow cursor position 
for the slice of interest. The shadow cursor position is output. 

[0007] In a further aspect, a computer program product for multi-modality 
registration using virtual cursors comprises a storage medium readable by a 
processing circuit and storing instructions for execution by the processing circuit for 
performing a method. The method comprises receiving a two-dimensional image 
dataset for an object at a first position and receiving a three-dimensional image 
dataset for the object at the first position. The three-dimensional image dataset 
includes a plurality of image slices. The two-dimensional image dataset is registered 
with the three-dimensional image dataset without taking into account a magnification 
factor. A user cursor position for a location in the two-dimensional image dataset is 
received. A slice of interest in the three-dimensional image dataset is received. The 
slice of interest is selected from the plurality of image slices in the three-dimensional 
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image dataset. A shadow cursor position for a location in the three dimensional 
dataset is calculated. The shadow cursor position corresponds to the user cursor 
position and the calculating includes a correction for the magnification factor 
corresponding to the shadow cursor position for the slice of interest. The shadow 
cursor position is output. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0008] Referring to the exemplary drawings wherein like elements are 
numbered alike in the several Figures: 

[0009] FIG. 1 is a pictorial view of a medical imaging system for performing 
multi-modality imaging; 

[0010] FIG. 2 depicts a reference coordinate system utilized by exemplary 
embodiments of the present invention; 

[001 1] FIG. 3 depicts x-ray and ultrasound coordinate systems; 

[0012] FIG. 4 is a flow diagram of an exemplary process for creating a 
shadow cursor; 

[0013] FIG. 5 is a flow diagram of an exemplary calculation for deriving an 
ultrasound coordinate location based on a x-ray coordinate location; 

[0014] FIG. 6 is a summary of the equations utilized to perform the 
calculation described in reference to FIG. 5; and 

[0015] FIG. 7 is a sample display of an x-ray image next to an ultrasound 
image in accordance with exemplary embodiments of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

[0016] Disclosed herein is a multi-modality registration technique using 
virtual cursors with rigid registration. Utilizing exemplary embodiments of the 
present invention, x-ray data and ultrasound data may be registered to each other, 
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including taking into account a magnification factor, while keeping the original data 
size and preserving the normal appearance of the two modalities of data. The virtual 
cursor registration technique eliminates the need to scale up the acquired data, thus 
reducing the required memory to handle large amounts of data. This speeds up the 
multi-modality registration. In addition the virtual cursor registration technique 
retains the data as it was acquired and therefore keeps the natural appearance of the 
data. Exemplary embodiments of the present invention may be utilized with multi- 
modality mammography as well as any other multi-modality imaging technologies 
where translation between the multiple modalities is desired. 

[0017] FIG. 1 is a pictorial view of a medical imaging system 12 for 
performing multi-modality imaging. Image data collected by the medical imaging 
system 12 may be input to exemplary embodiments of the present invention for 
simultaneous viewing. In an exemplary embodiment of the present invention, the 
imaging system 12 includes an ultrasound imaging system 14, a probe mover 
assembly 16, an ultrasound probe 18, and an x-ray imaging system 20. The 
ultrasound imaging system 14, probe mover assembly 16, ultrasound probe 18 and x- 
ray imaging system 20 may be operationally integrated in the imaging system 12 or 
they may be physically integrated in a unitary imaging system 12. FIG. 1 also 
includes a compression paddle 56 that is installed in the x-ray imaging system 20 
through a compression paddle receptacle 100. The probe mover assembly 16 may be 
attached to a receptacle (not shown) on a plurality of guide rails (not shown) on an x- 
ray positioner 102, above the compression paddle receptacle 100 through an 
attachment 104. Alternatively, the probe mover assembly 16 may be attached using a 
plurality of side handrails on the x-ray imaging system 20. As depicted in FIG. 1, the 
ultrasound probe 18 is connected to the ultrasound imaging system 14 on one end, and 
interfaces with the probe mover assembly 16 through a probe receptacle 106. In 
addition, the imaging system depicted in FIG. 1 includes an x-ray tube housing 108 
and a radiation source 110. When FIG. 1 is utilized for mammography, a patient is 
placed adjacent the x-ray imaging system 20 with a breast positioned between 
compression paddle 56 and detector 26. 



5 



140306-1 



[0018] The ultrasound probe 18 and the probe mover assembly 16 geometry 
are calibrated with respect to the compression paddle 56. In one embodiment, 
calibrating the ultrasound probe 18 includes ensuring that the ultrasound probe 18 is 
installed into the probe mover receptacle 104, and that probe mover assembly 16 is 
attached to. the x-ray imaging system 20 through the compression paddle receptacle 
100. Calibrating the imaging system 12 facilitates ensuring that the transformation 
operations between coordinate systems are validated. The calibrating may be utilized 
to perform a mechanical registration between the different image modalities produced 
by the system depicted in FIG. 1 . A correct beam-forming code environment is 
installed on ultrasound imaging system 14 to facilitate correcting refractive effects 
through compression paddle 56. Optimal parameters are then determined based on a 
prior knowledge of the patient or previous x-ray or ultrasound examinations. 

[0019] The patient is positioned in a cranio-caudal, medial-lateral, or oblique 
position, such that the object of interest (e.g., a breast) is positioned between the 
compression paddle 56 and the detector 26. The compression paddle 56 is then used 
to compress the object of interest to an appropriate thickness using at least one of a 
manual control on the receptacle 100 and an automatic control for receptacle 100. X- 
rays for the object of interest are then taken by the x-ray imaging system 20 operating 
in a standard 2D mode. The results of the x-rays are stored in an x-ray image dataset 
and/or displayed on a device in communication with the x-ray imaging system 20. 

[0020] The ultrasound probe 18, vertically mounted above compression 
paddle 56, is electro-mechanically scanned over the entire object of interest to 
generate a 3D dataset of the object of interest. The real time ultrasound data may be 
viewed on a monitor of the ultrasound imaging system 14 or on any display in 
communication with the ultrasound imaging system 14. In addition, the volumetric 
ultrasound data may be stored in an ultrasound image dataset. 

[0021] FIG. 1 also includes a computer system 10 in communication with the 
ultrasound imaging system 14 and the x-ray imaging system 20. Communication may 
be via any network known the art (e.g., Internet, local area network). In an alternate 
exemplary embodiment of the present invention, the computer system 10 is physically 
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located within the imaging system 12 and a network may not be required for 
communication. The computer system 10 includes instructions for performing the 
processing described in reference to FIGS. 4 and 5 and may include storage for the 
image data produced by the ultrasound imaging system 14 and/or the x-ray imaging 
system 20. Any kind of computer system known in the art (e.g., personal computer, 
integrated circuit module, host computer) may be utilized by exemplary embodiments 
of the present invention. 

[0022] FIG. 2 depicts a reference coordinate system utilized by exemplary 
embodiments of the present invention. The reference coordinate system is the 
coordinate system being utilized by exemplary embodiments of the present invention 
as the base coordinate system for use in creating the correspondence between image 
datasets of different modalities. FIG. 2 includes a coordinate location for the x-ray 
source 202, which maybe expressed in the reference coordinate system as Ps(Xs, Ys, 
Zs), where Zs is always the same value because the x-ray image is 2D and therefore, 
the plane being covered is constant. FIG. 2 also shows the direction of the ultrasound 
probe movement 204, the compression plate 56, the placement of the object of interest 
(e.g., a breast) 206, the location of the patient's chest wall 208, and the location of the 
x-ray detector 26 all relative to the reference coordinate system. The reference 
coordinate system includes superior ("S" or "z"), anterior ("A" or "x") and left ("L" 
or "y") coordinates to specify locations in the reference coordinate system. 

[0023] FIG. 3 depicts an anterior left superior (ALS) x-ray coordinate system 
and an anterior inferior right (AIR) ultrasound coordinate system relative to the 
reference coordinate system 200 depicted in FIG. 2. In addition, FIG. 3 depicts the 
ALS x-ray coordinate system and AIR ultrasound coordinate system relative to the 
ultrasound and x-ray imaging systems. The ultrasound coordinate system 304 
depicted in FIG 3 includes anterior ("A), right ("R") and inferior ("I") coordinates for 
each voxel in the ultrasound image dataset. The x-ray coordinate system 306 depicted 
in FIG. 3 is expressed in terms of the same coordinate system as the reference 
coordinate system 200. 
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[0024] FIG. 4 is a flow diagram of an exemplary process for creating a 
shadow, or virtual, cursor. At step 402, the 2D x-ray image dataset is mechanically 
registered to the 3D ultrasound image dataset. This preliminary registration process at 
step 402 does not account for the magnification factor in the x-ray image data. The 
registration process may include a mechanical registration, as is known the art, to 
obtain acquisitions coordinate system for both dataset and how they transform into 
each other. The mechanical registration process may be used if the datasets have been 
acquired at the same time. If the subject is scanned at different times or moved during 
the scanning, then a registration type known in the art such as longitudinal registration 
may be utilized. 

[0025] At step 404, a visual depiction of the x-ray image is created from data 
contained in the x-ray image dataset and displayed on a display device. Also 
displayed on the display device, adjacent to the x-ray image, is an ultrasound image 
created from the data contained in the ultrasound image dataset. In exemplary 
embodiments of the present invention, the x-ray data is displayed in a lateral position 
(e.g., from right to left) and the ultrasound data is displayed in a sagittal position (e.g., 
from top to bottom). See FIG. 7 for an alternate exemplary embodiment of the 
present invention that includes a display arrangement with one x-ray view and three 
ultrasound views within a single display. On the left is the x-ray image 702 with the 
current location of the user cursor denoted by an arrow within the x-ray image 702. 
The ultrasound image includes a sagittal view 704 for a particular slice within the 
ultrasound image with a shadow, or virtual, cursor denoted by an arrow. In addition, 
the ultrasound image depicted in FIG. 7 includes an axial view 708 and a coronal 
view 706 of the slice. 

[0026] Referring to step 406 in FIG. 4, a user cursor is detected at a particular 
location in the x-ray image. In response to detecting the user cursor location, step 408 
is performed. In step 408, a linear equation, such as the one described in reference to 
FIG. 5, is applied to the x-ray image location where the user cursor is located. Inputs 
to the equation include the particular ultrasound slice being currently viewed as well 
as the user cursor location in the x-ray image. Applying the linear equation results in 
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a corresponding location in the ultrasound image that has been corrected for 
magnification. 

[0027] At step 410, a shadow, or virtual, cursor is moved to the corresponding 
ultrasound image location. The location of the shadow cursor is visible to the user 
who is viewing the images on the display. The loop from step 406 through 410 is 
performed until the user exits out of the images and/or the application. Performing 
the loop from step 406 through step 410 allows the shadow cursor to move around the 
ultrasound image in a manner that is simultaneous or nearly simultaneous to the 
movement of the user cursor in the x-ray image. Alternate exemplary embodiments 
of the present invention input a user cursor location from an ultrasound image and 
calculate a corresponding shadow cursor location on an x-ray image. 

[0028] FIG. 5 is a flow diagram of an exemplary calculation for deriving an 
ultrasound coordinate location based on an x-ray coordinate location. At step 502, an 
x-ray source coordinate, Ps, is received. Ps specifies the point in the x-ray coordinate 
system 306 where the x-ray source is located and is specified as Ps(Xs, Ys, Zs). Next, 
at step 504, a user cursor location in an x-ray image dataset is received and specified 
in the x-ray coordinate system 306 as Px(Xx, Yx, Zx). Processing continues at step 
506, where the ultrasound slice that the user is viewing or that the user requests to 
view is received. The ultrasound location corresponding to the x-ray location may be 
specified as Pu(Xu, Yu, Zu). When the ultrasound slice is specified, the Zu 
component of the ultrasound location is assigned to the slice value. 

[0029] Next, at step 508, a magnification correction for the x-ray location is 
derived based on the data received in steps 502, 504 and 506. In exemplary 
embodiments of the present invention, the magnification correction may be derived 
based on the following formula: t = (Zu-Zs)/(Zx-Zs). After the magnification 
correction is derived, step 510 is performed to calculate the ultrasound coordinate Xu. 
Xu may be calculated as: Xs + (Xx - Xs)(t). By using the calculation at step 510, the 
resulting Xu coordinate includes an adjustment for magnification. The Xy coordinate 
is calculated at step 5 12 as: Ys + (Yx - Ys)(t). The resulting corresponding 
ultrasound location, which has been adjusted for magnification, is then output at step 
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514. In exemplary embodiments of the present invention, the output is to the shadow 
cursor processing described in reference to FIG. 4. FIG. 6 is a summary of the 
equations utilized to perform the calculation described in reference to FIG. 5 as well 
as exemplary derivations of the formulas. 

[0030] Alternate exemplary embodiments of the present invention include 
receiving an ultrasound coordinate point from a user cursor and then determining a 
corresponding shadow cursor location for in an x-ray image. In addition, other types 
of modalities may be utilized with exemplary embodiments of the present invention 
and more than one shadow cursor may be created based on receiving the coordinates 
of a user cursor. 

[0031] Exemplary embodiments of the present invention have been described 
in reference to apparatuses and methods for mammography. It should be appreciated, 
however, that the teachings of the present invention may also be utilized in other 
areas, such as lung imaging, brain imaging, liver imaging, kidney imaging, bone 
imaging and other medical areas, as well as in industrial applications, such as 
detecting low density regions in fabricated parts, or performing fault/fatigue testing 
(e.g., examining for cracks, depressions, or impurities). 

[0032] Exemplary embodiments of the present invention may be utilized to 
more quickly view and compare images collected using different imaging techniques. 
The use of a linear equation allows the conversion between two or more modalities to 
be performed quickly and without requiring that large amounts of registration data be 
stored. Using a linear equation allows the conversion to be performed quickly enough 
for the shadow cursor and user cursor to appear to be moving simultaneously, or near 
simultaneously on a display device that is displaying images created from different 
modalities. In addition, the 3D image (e.g., ultrasound) has the same appearance that 
it had when it was acquired. This is accomplished while still having a pixel/voxel 
correspondence mechanism between 3D (e.g., ultrasound) and 2D (e.g., x-ray) 
images. As described previously, exemplary embodiments of the present invention 
may be applied to other multi-modality data acquisitions. 
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[0033] As described above, embodiments can be embodied in the form of 
computer-implemented processes and apparatuses for practicing those processes. In 
exemplary embodiments, the invention is embodied in computer program code 
executed by one or more network elements. Embodiments include computer program 
code containing instructions embodied in tangible media, such as floppy diskettes, 
CD-ROMs, hard drives, or any other computer-readable storage medium, wherein, 
when the computer program code is loaded into and executed by a computer, the 
computer becomes an apparatus for practicing the invention. Embodiments include 
computer program code, for example, whether stored in a storage medium, loaded 
into and/or executed by a computer, or transmitted over some transmission medium, 
such as over electrical wiring or cabling, through fiber optics, or via electromagnetic 
radiation, wherein, when the computer program code is loaded into and executed by a 
computer, the computer becomes an apparatus for practicing the invention. When 
implemented on a general-purpose microprocessor, the computer program code 
segments configure the microprocessor to create specific logic circuits. 

[0034] While the invention has been described with reference to exemplary 
embodiments, it will be understood by those skilled in the art that various changes 
may be made and equivalents may be substituted for elements thereof without 
departing from the scope of the invention. In addition, many modifications may be 
made to adapt a particular situation or material to the teachings of the invention 
without departing from the essential scope thereof. Therefore, it is intended that the 
invention not be limited to the particular embodiment disclosed as the best mode 
contemplated for carrying out this invention, but that the invention will include all 
embodiments falling within the scope of the appended claims. Moreover, the use of 
the terms first, second, etc. do not denote any order or importance, but rather the 
terms first, second, etc. are used to distinguish one element from another. 
Furthermore, the use of the terms a, an, etc. do not denote a limitation of quantity, but 
rather denote the presence of at least one of the referenced item. 



11 



